Image processing apparatus, image processing method, and program

ABSTRACT

There is provided an image processing apparatus including a dynamic body detecting unit for detecting a dynamic body contained in a moving image, a dynamic body region setting unit for, during a predetermined time from a time point the dynamic body is detected by the dynamic body detecting unit, setting a region containing the dynamic body at the detection time point as a dynamic body region, and a fluctuation removable processing unit for performing a fluctuation removal process on a region other than the dynamic body region set by the dynamic body region setting unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 12/610,772 filed Nov. 2, 2009, which claims the benefit of priorityfrom the prior Japanese Patent Application No. 2008-302279, filed Nov.27, 2008. The entire contents of each of the above applications areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and a program.

2. Description of the Related Art

When photographing a moving image with a video camera and the like,fluctuation may occur in the photographed image due to the influence offluctuation of air. Such fluctuation of air is thought to occur when theindex of refraction of light locally changes due to temperaturedifference and the like. When photographing a distant view using atelescopic lens, the light path becomes long as the distance from thesubject to the video camera is large. Thus, when using the telescopiclens, the photographed image may be greatly influenced by thefluctuation of air and a noticeable fluctuation may appear in thephotographed image. Specifically, in the image subjected to theinfluence of fluctuation, part of the subject that is originally linearappears undulating or a portion that appears distorted may be contained.

Such influence of fluctuation appears as a phenomenon in which thecoordinate position of each pixel configuring the image varies within alocal region. Thus, the influence of fluctuation can be removed bycorrecting the coordinate position of each pixel in the image. Forinstance, the influence of fluctuation with respect to the stationarysubject can be easily removed by combining a filtering process such asan averaging process and a median process. However, the image degradesif the filtering process is performed on the moving subject rather thanremoving the influence of fluctuation. With regards to such issue,Japanese Patent Application Laid-Open No. 2008-160733 discloses atechnique of detecting the moving subject using a dynamic body detectingtechnique, and performing the filtering process only on the stationarysubject. Through the use of such technique, the filtering process willnot be performed on an image region of the moving subject, anddegradation of the image by such filtering process can be prevented.

SUMMARY OF THE INVENTION

The technique disclosed in Japanese Patent Application Laid-Open No.2008-160733 is based on the assumption that the moving subject can becorrectly detected by the dynamic body detecting process. However, themoving subject may not actually be correctly detected as the dynamicbody by the dynamic body detecting process. Thus, the filtering processmight be performed on the moving subject, whereby part of the movingsubject may disappear or the image quality may degrade only at therelevant portion.

The present invention has been made in view of the above issues, and itis desirable to provide a novel and improved image processing apparatus,an image processing method, and a program capable of avoiding the imagequality from degrading due to the fluctuation removal process even ifpart of the moving subject is not correctly recognized as the dynamicbody.

According to an embodiment of the present invention, there is providedan image processing apparatus including a dynamic body detecting unitfor detecting a dynamic body contained in a moving image, a dynamic bodyregion setting unit for, during a predetermined time from a time pointthe dynamic body is detected by the dynamic body detecting unit, settinga region containing the dynamic body at the detection time point as adynamic body region, and a fluctuation removable processing unit forperforming a fluctuation removal process on a region other than thedynamic body region set by the dynamic body region setting unit.

The fluctuation removal processing unit may perform other fluctuationremoval process, in which a video of the dynamic body region containedin the dynamic body region does not disappear, on part of or the entiredynamic body region set by the dynamic body region setting unit.

The image processing apparatus may further include a super-resolutionprocessing unit for performing a super-resolution process on part of orthe entire dynamic body region not performed with the other fluctuationremoval process by the fluctuation removal processing unit of thedynamic body region set by the dynamic body region setting unit.

The fluctuation removal process performed on an entire region of themoving image excluding the dynamic body region by the fluctuationremoval processing unit may be an averaging process or a median processof a pixel value contained in each frame of the moving image.Furthermore, the fluctuation removal process performed on the dynamicbody region may be a fluctuation removal process based on a non-rigidbody registration.

According to another embodiment of the present invention, there isprovided An image processing method including the steps of detecting adynamic body contained in a moving image, setting, during apredetermined time from a time point the dynamic body is detected in thedynamic body detecting step, a region containing the dynamic body at thedetection time point as a dynamic body region, and performing afluctuation removal process on an entire region of the moving imageexcluding the dynamic body region set in the dynamic body region settingstep.

According to another embodiment of the present invention, there isprovided a program for causing a computer to execute a dynamic bodydetecting function for detecting a dynamic body contained in a movingimage, a dynamic body region setting function for, during apredetermined time from a time point the dynamic body is detected by thedynamic body detecting function, setting a region containing the dynamicbody at the detection time point as a dynamic body region, and afluctuation removable processing function for performing a fluctuationremoval process on an entire region of the moving image excluding thedynamic body region set by the dynamic body region setting function.

According to the embodiments of the present invention described above,even if part of the moving subject is not correctly recognized as adynamic body, the image quality can be avoided from degrading due to thefluctuation removal process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory view describing the influence that arises whena filtering process is performed on a moving subject;

FIG. 2 is an explanatory view showing an example of a functionconfiguration of an image processing apparatus according to oneembodiment of the present invention;

FIG. 3 is an explanatory view showing one example of a high resolutionprocessing method according to the embodiment;

FIG. 4 is an explanatory view showing an overall flow of imageprocessing according to the present embodiment;

FIG. 5 is an explanatory view showing a flow of image processingaccording to the present embodiment;

FIG. 6 is an explanatory view showing an example of an image processingmethod according to the embodiment; and

FIG. 7 is an explanatory view showing an example of a hardwareconfiguration of an information processing apparatus capable ofrealizing the image processing method according to the embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENT

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in the specification and the appended drawings, structural elements thathave substantially the same function and structure are denoted with thesame reference numerals, and repeated explanation of these structuralelements is omitted.

[Regarding Flow of Description]

The flow of description related to the embodiment of the presentinvention described below will be briefly described. First, withreference to FIG. 1, the issues that arise by performing the fluctuationremoval process based on the averaging process, the median process andthe like will be briefly described. A function configuration of an imageprocessing apparatus 100 according to one embodiment of the presentinvention will then be described with reference to FIG. 2.

A super-resolution process, which is one example of a high resolutionprocess according to the embodiment, will be schematically describedwith reference to FIG. 3. The flow of image processing according to theembodiment will be described with reference to FIGS. 4 and 5. A methodof applying the image processing according to the embodiment to ablock-divided region will be described with reference to FIG. 6. Anexample of a hardware configuration of an information processingapparatus capable of realizing the functions of the image processingapparatus 100 according to the embodiment will be briefly described withreference to FIG. 7. Lastly, the technical idea of the embodiment willbe summarized, and the effects obtained from such technical idea will bebriefly described.

(Describing Items)

1: Organization of issues (Regarding fluctuation removal process byfiltering)

2: Embodiment

-   -   2-1: Regarding function configuration of image processing        apparatus 100    -   2-2: Regarding non-rigid body registration processing technique    -   2-3: Regarding super-resolution processing technique    -   2-4: Regarding flow of image processing    -   2-5: Regarding method of reducing calculation load by block        division    -   2-6: Regarding example of hardware configuration of image        processing apparatus 100    -   2-7: Summary

<1: Organization of Issues>

First, prior to describing the technique according to one embodiment ofthe present invention in detail, the issues to be solved by theembodiment will be briefly described with reference to FIG. 1.

FIG. 1 shows a fluctuation removal method based on the averagingprocess. As already described earlier, the influence of fluctuationsometimes appear in the photographed image due to the influence of thefluctuation of air when imaging a moving image with a video camera orthe like. Such influence of fluctuation can be removed by performing thefiltering process such as the averaging process and the median processon the photographed image. However, if the filtering process isperformed on the image region containing the moving subject (hereinafterreferred to as dynamic body), a portion of the dynamic body maydisappear, or the image quality may degrade (residual image etc.) at therelevant portion.

For instance, when a moving automobile is photographed, the position ofthe automobile changes every second in a plurality of photographedimages (hereinafter referred to as time-series images) configuring themoving image. Thus, if the pixel values of the image region containingthe automobile are averaged for the time-series images, an average valueis calculated with respect to the pixel value of a background imagerecorded after the movement of the automobile, whereby the contour orthe color of the automobile fades as a result. That is, the automobiledisappears from the photographed image due to the averaging filteringprocess. Such phenomenon similarly occurs for the median process. Atechnique of performing the filtering process only on the image regionnot containing the dynamic body is thus developed.

This technique detects the dynamic body from the photographed image, andperforms the filtering process on the image region other than the regionin which the dynamic body is detected. However, if the dynamic body isnot correctly detected, only the dynamic body portion that is notdetected disappears, or the image quality degrades. Such failure in thedynamic body detection tends to occur in the image region containing theback portion of a truck etc. The back portion of the truck is often longin the moving direction and is a single color or a monotonous color. Thedynamic body detecting process calculates the difference of the pixelvalue for every pixel or for every image region among a plurality ofphotographed images of the time-series images, and determines the sameas the pixel or the image region containing the dynamic body if thedifference value is greater than a predetermined threshold value.

Therefore, if the pixel value is not changed in the pixel or the imageregion of the photographed image to be subjected to the differencecalculation, the pixel or the image region may not be determined as thedynamic body although it is actually the dynamic body. FIG. 1 shows theback portion of the truck by way of example. In FIG. 1, time-seriesimages P1 to P6 and an average image after performed with the averagingfiltering process are schematically shown. Photographed images P1, P2included in the time-series images P1 to P6 are separately shown. Thephotographed image P1 is recorded earlier than the photographed imageP2.

First, attention is to be made on the photographed images P1, P2. Thedynamic body detecting process focuses on a certain pixel or an imageregion (hereinafter referred to as reference region) and determineswhether or not a pixel value of the reference region differs by greaterthan or equal to a predetermined value among a plurality of photographedimages, as described above. For example, reference regions R1, R1′ shownin FIG. 1 are to be focused. The reference region R1 of the photographedimage P1 corresponds to the reference region R1′ of the photographedimage P2. In the photographed image P1, the reference region R1 containsthe driver seat portion of the truck. However, the reference region R1′of the photographed image P2 does not contain the driver seat portion ofthe truck as the truck is moving at the time point the photographedimage P2 is recorded. Thus, a large difference creates between the pixelvalue of the reference region R1 and the pixel value of the referenceregion R1′. As a result, the subject contained in the reference regionsR1, R1′ is determined as the dynamic body.

Reference regions R2, R2′ shown in FIG. 1 are then to be focused. Thereference region R2 of the photographed image P1 corresponds to thereference region R2′ of the photographed image P2. In the photographedimage P1, the reference region R2 contains the back portion of thetruck. The truck is moving at the time point photographed image P2 isrecorded, as described above. However, the reference region R2′ of thephotographed image P2 still contains the back portion of the truck. Asmentioned above, the back portion of the truck often has a monotonouscolor. Thus, a large difference does not create between the pixel valueof the reference region R2 and the pixel value of the reference regionR2′ even if the portion that is actually different is photographed. As aresult, the subject contained in the reference regions R2, R2′ isdetermined as the non-dynamic body.

The average image shown in FIG. 1 will now be focused. The driver seatportion of the truck etc. is determined as the dynamic body, and theback portion is determined as the non-dynamic body. Thus, the backportion of the truck disappears from the average image if the averagingfiltering process is performed on the non-dynamic body portion.Therefore, determination may not be made as the dynamic body in thedynamic body detecting process although it is actually the dynamic body.As a result, the relevant portion disappears after the filtering processor the image quality degrades, whereby the photographed image becomesunnatural and odd. In the embodiment to be hereinafter described, thefluctuation removal method in which, even if part of the dynamic body isdetermined as the non-dynamic body, the part of the dynamic body doesnot disappear or the image quality does not degrade is proposed.

<2: Embodiment>

One embodiment of the present invention will be described. The presentembodiment assumes an image region (hereinafter referred to as dynamicbody region) containing the dynamic body as the dynamic body region fora predetermined time when the dynamic body is detected, and prevents thefluctuation removal process from being performed on the relevant regioneven after the dynamic body has moved.

[2-1: Regarding Function Configuration of Image Processing Apparatus100]

First, the function configuration of the image processing apparatus 100according to the present embodiment will be described with reference toFIG. 2. FIG. 2 is an explanatory view showing an example of the functionconfiguration of the image processing apparatus 100 according to thepresent embodiment.

As shown in FIG. 2, the image processing apparatus 100 mainly includesan image input unit 102, a dynamic body detecting unit 104, a dynamicbody region setting unit 106, a timer 108, an operation input unit 110,a first fluctuation removing unit 112, a second fluctuation removingunit 114, and a high image quality processing unit 116. The imageprocessing apparatus 100 may also include an imaging unit (not shown)for imaging an image of a subject.

First, the image input unit 102 is input with a photographed image. If amoving image is input, the image input unit 102 is input withtime-series images configured by a plurality of photographed images. Thephotographed image input to the image input unit 102 is input to thedynamic body detecting unit 104 and the first fluctuation removing unit112. When the photographed image is input from the image input unit 102to the dynamic body detecting unit 104, the dynamic body detecting unit104 executes the dynamic body detecting process based on the inputphotographed image.

For instance, the dynamic body detecting unit 104 calculates adifferential pixel value of the photographed image taken temporallybefore and after for each image region of the photographed image of thetime-series images. The dynamic body detecting unit 104 also determineswhether or not the calculated differential pixel value is greater than apredetermined threshold value, and determines the image region greaterthan the predetermined threshold value as the dynamic body. In thefollowing description, the image region determined as the dynamic bodyby the dynamic body detecting unit 104 is sometimes referred to as thedynamic body region. Each image region may be set in units of pixels, ormay be set in units of pixel blocks configured by a predetermined numberof pixels. The number of photographed images referenced by the dynamicbody detecting unit 104 for the dynamic body detection is not limited totwo, and may be three or more. Moreover, the dynamic body detectingmethod of setting a sufficiently large number of photographed images tobe referenced so that the subject that fluctuates at a constant periodsuch as tree or wave is not detected as the dynamic body may be used incombination.

The information of the dynamic body region detected by the dynamic bodydetecting unit 104 is input to the dynamic body region setting unit 106.The dynamic body region detected by the dynamic body detecting unit 104does not include a subject of monotonous color as illustrated by theback portion of the truck etc. Therefore, a new dynamic body region isto be set such that the dynamic body region includes the image regioncontaining such subject. The dynamic body region setting unit 106 thusnewly sets the dynamic body region and the non-dynamic body region basedon the information of the dynamic body region input from the dynamicbody detecting unit 104. The dynamic body region setting unit 106 doesnot newly detect part of the dynamic body not detected by the dynamicbody detecting unit 104, and maintains the dynamic body region detectedby the dynamic body detecting unit 104 as the dynamic body region for apredetermined time.

With reference to the example of FIG. 1, for example, the referenceregion R1 of the photographed image P1 is judged as the dynamic bodyregion by the dynamic body detecting unit 104. The dynamic body regionsetting unit 106 assumes the image region corresponding to the referenceregion R1 as the dynamic body region for a predetermined time. Asdescribed above, the reference region R2 of the photographed image P1 isdetermined as the non-dynamic body region by the dynamic body detectingunit 104 as the difference with the reference region R2′ of thephotographed image P2 is small. However, the image region (referenceregion R1′) corresponding to the reference region R1 is assumed as thedynamic body region for a predetermined time by the dynamic body regionsetting unit 106. That is, the front side of the back portion of thetruck corresponding to the reference region R1′ is set as the dynamicbody region. For instance, the entire back portion of the truck is setas the dynamic body region by setting the time for the entire backportion of the truck to pass through the image region corresponding tothe reference region R1 as the predetermined time.

The predetermined time is appropriately set according to the subject tobe photographed etc. The predetermined time is set by the timer 108. Thetimer 108 starts counting of time in response to an instruction to startcounting received from the dynamic body region setting unit 106, andnotifies the dynamic body region setting unit 106 that a predeterminedtime has elapsed at the time point the predetermined time has elapsed.The dynamic body region setting unit 106 uses the timer 108 to set theimage region corresponding to the dynamic body region as the dynamicbody region in the photographed image until the time point thepredetermined time has elapsed with the photographed image containingthe dynamic body region detected by the dynamic body detecting unit 104as a reference. That is, the image region corresponding to the dynamicbody region of the photographed image that becomes a reference is set asthe dynamic body region of the photographed image in the time-seriesimages photographed within a predetermined time after such photographedimage. The image region not set as the dynamic body region by thedynamic body region setting unit 106 is set as the non-dynamic bodyregion.

The information of the non-dynamic body region set by the dynamic bodyregion setting unit 106 is input to the first fluctuation removing unit112. The information of the dynamic body region set by the dynamic bodyregion setting unit 106 is input to the second fluctuation removing unit114. Furthermore, if the image region is specified using the operationinput unit 110, the dynamic body region setting unit 106 sets therelevant image region as a specified region, and inputs the informationof the specified region to the high image quality processing unit 116.The specified region will be hereinafter described.

As described above, the first fluctuation removing unit 112 is inputwith the photographed image input to the image input unit 102, and theinformation of the non-dynamic body region set by the dynamic bodyregion setting unit 106. The first fluctuation removing unit 112performs a fluctuation removal process on the non-dynamic body region ofthe photographed image input from the image input unit 102 based on theinformation of the non-dynamic body region input from the dynamic bodyregion setting unit 106. The fluctuation removal process performed onthe photographed image by the first fluctuation removing unit 112 is,for example, a filtering process (hereinafter referred to as firstfluctuation removal process) such as the averaging process and themedian process. As described earlier, if the non-dynamic body regiondetermined in the dynamic body detecting process contains the dynamicbody, the first fluctuation removal process is performed on such dynamicbody, whereby the dynamic body may disappear from the output image orthe image of the dynamic body portion may degrade.

In the present embodiment, however, part of the dynamic body, which isactually a dynamic body but is not detected in the dynamic bodydetecting process, is excluded from the non-dynamic body region by theprocess of the dynamic body region setting unit 106. Thus, the firstfluctuation removal process will not be performed on the part of thedynamic body in the non-dynamic body region determined in the dynamicbody detecting process of the dynamic body detecting unit 104, and partof the dynamic body does not disappear from the output image. It can berecognized that the part of the dynamic body that is not excluded fromthe non-dynamic body region in the process of the dynamic body regionsetting unit 106 sometimes disappears from the output image of the firstfluctuation removal process or the image sometimes degrades. In mostcases, however, sufficient effects are achieved by the technique of thepresent embodiment.

The photographed image (hereinafter referred to as first processedimage) performed with the first fluctuation removal process by the firstfluctuation removing unit 112 is input to the second fluctuationremoving unit 114. The second fluctuation removing unit 115 is alsoinput with the information of the dynamic body region from the dynamicbody region setting unit 106. As described above, the first processedimage is only performed with the first fluctuation removal process onthe non-dynamic body region. Thus, a fluctuation removal process(hereinafter referred to as second fluctuation removal process), withwhich the dynamic body does not disappear or the image does not degradein the dynamic body region, is performed on the dynamic body region ofthe first processed image by the second fluctuation removing unit 114.An image processing technique called a non-rigid body registration isused for the second fluctuation removal process. The non-rigid bodyregistration processing technique will be hereinafter described. Thesecond fluctuation removing unit 114 may be configured to perform thesecond fluctuation removal process on the dynamic body region specifiedthrough the operation input unit 110.

The photographed image (hereinafter referred to as second processedimage) performed with the second fluctuation removal process by thesecond fluctuation removing unit 114 is input to the high image qualityprocessing unit 116. The first processed image or the second processedimage may be displayed on a display screen (not shown) after thefluctuation removal process is performed by the first fluctuationremoving unit 112 or the second fluctuation removing unit 114. Accordingto such configuration, the user can operate the operation input unit 110and specify the specifying region while referencing the first processedimage or the second processed image displayed on the display screen. Thehigh image quality processing unit 116 performs a high image qualityprocessing on the specified region specified through the operation inputunit 110 of the image regions in the second processed image.

A very high calculation processing ability is normally desired torealize high image quality processing. Thus, the image processingapparatus 100 is configured to perform high image quality processingonly on the specified region instead of performing the high imagequality processing on the entire image region in the photographed image.It can be recognized that the high image quality process may beperformed on the entire image region of the photographed image if theimage processing apparatus 100 is realized by hardware having asufficiently high calculation processing ability. A super-resolutionprocessing technique, and the like are used for the high image qualityprocessing. The super-resolution technique will be hereinafterdescribed. The photographed image (hereinafter referred to as processedimage) performed with the high image quality processing by the highimage quality processing unit 116 is then output to the outside of theimage processing apparatus 100.

The function configuration of the image processing apparatus 100according to the present embodiment has been described above. Theconfiguration described herein is an example, and a variant in which thesecond fluctuation removing unit 114 and the high image qualityprocessing unit 116 are omitted may be adopted. If the secondfluctuation removing unit 114 is omitted, the first processed imageoutput from the first fluctuation removing unit 112 is input to the highimage quality processing unit 116. If the high image quality processingunit 116 is omitted, the operation input unit 110 is also omitted, andthe second processed image output from the second fluctuation removingunit 114 is output as the processed image. Furthermore, if the secondfluctuation removing unit 114 and the high image quality processing unit116 are omitted, the first processed image output from the firstfluctuation removing unit 112 is output to the outside as the processedimage.

The flow of image processing related to the first fluctuation removalprocess may be modified as below. In the description of FIG. 2, a flowin which the dynamic body region and the non-dynamic body region are setin advance by the dynamic body region setting unit 106, and the firstfluctuation removal process is performed by the first fluctuationremoving unit 112 based on the set content is shown. However, the firstfluctuation removal process may be performed in advance by the firstfluctuation removing unit 112 on the non-dynamic body region detected bythe dynamic body detecting unit 104, and the pixel value of the firstprocessed image and the pixel value of the original image may beappropriately selected by the dynamic body region setting unit 106 togenerate the output image (processed image).

In this case, the first fluctuation removing unit 112 performs the firstfluctuation removal process on the dynamic body region detected by thedynamic body detecting unit 104 and not on the dynamic body region resetby the dynamic body region setting unit 106. The dynamic body regionsetting unit 106 not only resets the dynamic body region and thenon-dynamic body region using the timer 108, but also generates theprocessed image with the pixel of the first processed imagecorresponding to the reset non-dynamic body region and the originalimage corresponding to the reset dynamic body region. Therefore, whenperforming such modification, the information of the non-dynamic bodyregion detected by the dynamic body detecting unit 104 is input to thefirst fluctuation removing unit 112. The dynamic body region settingunit 106 is input with the first processed image based on the detectionresult of the dynamic body detecting unit 104 output from the firstfluctuation removing unit 112, and the original image input to the imageinput unit 102. The first processed image generated by the dynamic bodyregion setting unit 106 is input to the second fluctuation removing unit114. In such modification, the functions of the second fluctuationremoving unit 114 and the high image quality processing unit 116 are notmodified.

As described above, the configuration of the image processing apparatus100 can be appropriately changed while maintaining the technicalfeatures of the present embodiment. The technical features of thepresent embodiment lie in that the dynamic body region is reset asdescribed above for all the frames included from the frame (photographedimage) in which the dynamic body is detected by the dynamic bodydetecting unit 104 to the frame of after elapse of a predetermined time.Through such resetting, the disappearance of the dynamic body from theprocessed image and the degradation of the image at the dynamic bodyportion due to the fluctuation removal process can be prevented.

[2-2: Regarding Non-Rigid Body Registration Processing Technique]

The non-rigid body registration process executed by the secondfluctuation removing unit 114 will be briefly described.

First, the registration used in the field of image processing oftenrefers to a method of estimating a conversion parameter the images matchwhen two images are overlapped. The method of estimating the conversionparameter assuming the photographing object is a non-rigid body issometimes particularly referred to as a non-rigid body registration. Inthe non-rigid body registration process, the control point thatpartitions the image space at a predetermined interval is set as aparameter, and the position of the control point where a predeterminedregistration evaluation function becomes a maximum is determined whilechanging the parameter. Furthermore, a deformation vector field of theimage space is estimated based on the determined position of the controlpoint.

A function containing the evaluation component for evaluating thesimilarity of the images to compare is used for the registrationevaluation function. The similarity evaluated by the registrationevaluation function includes an entropy correlation coefficient,Pearson's product-moment correlation coefficient, and the like. Thesimilarity is calculated based on the simultaneous distribution ofmulti-dimensional vector configured by the control point of each imageto compare. The deformation vector field of the image space is expressedby N−1 Bezier curves defined by N control points.

Therefore, the non-rigid body registration refers to the method ofestimating the deformation vector field of the image space as adeformation parameter. The above description is obviously schematic, andthe non-rigid body registration processing technique including numerousdevelopmental deformations is actually developed. It can be recognizedthat various non-rigid body registration processing techniques includingsuch developmental deformation can be used in the second fluctuationremoving unit 114 of the present embodiment.

[2-3: Regarding Super-Resolution Processing Technique]

The super-resolution process, which is an example of the high imagequality processing according to the present embodiment, will be brieflydescribed with reference to FIG. 3. FIG. 3 is an explanatory viewschematically showing the content of the super-resolution process.

As shown in FIG. 3, the super-resolution processing technique calculatesa high resolution image of relatively high resolution using thetime-series images (e.g., P10 to P14) of relatively low resolution. Thatis, the super-resolution process is a process of interpolating the highfrequency component using a plurality of low resolution images notcontaining the high frequency component, and reconfiguring the highresolution image containing the high frequency component. For instance,two images are photographed with the position shifted by ½ pixel withthe imaging sensor of the same resolution as the low resolution image,and the pixel values extracted from such images are alternately combinedone pixel at a time for synthesis. The high frequency component can berestored by performing such process. Instead of movement controlling theposition of the imaging sensor, the high frequency component can berestored using the position shift of smaller than one pixel caused byinstability etc. in time of photographing.

The super-resolution process is roughly divided to a position detectingprocess for detecting the relative position shift between the lowresolution images of the time-series images, and an image synthesizingprocess of synthesizing the low resolution images in view of theposition shift. In the example of FIG. 3, the component corresponding tothe position detecting process is noted as a position detecting unit134, and the component corresponding to the image synthesizing processis noted as an image synthesizing unit 132. In the position detectingunit 134, the amount of shift of the sampled position between each lowresolution image in the time-series images is detected at an accuracy ofsmaller than or equal to the pixel. The amount of shift of the sampledposition detected by the position detecting unit 134 is then input tothe image synthesizing unit 132. The image synthesizing unit 132combines and synthesizes the pixel values of the sampled positioncorresponding to each other based on the amount of shift of the sampledposition detected by the position detecting unit 134 to calculate thepixel value of the reconfigured image (high resolution image).

In the example of FIG. 3, a target image RP of low resolution formedwith pixels PX1 is reconfigured to a target image RP′ of high resolutionformed with pixels PX1′ by executing the super-resolution process. Thesuper-resolution process is a process of a very high calculation load.Thus, the process may not finish within a realistic time depending onthe number of low resolution images to be referenced in thereconfiguration process or the number of low resolution images to beprocessed, or the size of the image region to be subjected to thesuper-resolution process. Thus, the image processing apparatus 100according to the present embodiment is configured to perform thesuper-resolution process only on the image region specified through theoperation input unit 110.

The super-resolution processing technique has been schematicallydescribed above. With respect to such super-resolution processingtechnique, various researches and developments are being made toefficiently and accurately execute the process corresponding to theposition detecting unit 134 and the process corresponding to the imagesynthesizing unit 132. An arbitrary technique achieved through suchresearches and developments can be used in combination with thetechnique of the present embodiment.

[2-4: Regarding Flow of Image Processing]

The flow of image processing according to the present embodiment willnow be described with reference to FIGS. 4 and 5. FIG. 4 is anexplanatory view showing an overall flow of image processing accordingto the present embodiment. FIG. 5 is an explanatory view showing a flowrelated to a setting process of a dynamic body region in the overallflow shown in FIG. 4.

The image processing described here is realized using the imageprocessing apparatus 100 shown in FIG. 2. In the image processing methodshown in FIGS. 4 and 5, the fluctuation removal process is executedthrough a procedure slightly different from the flow of image processingin the image processing apparatus 100 shown in FIG. 2. Morespecifically, the difference lies in that in the relevant imageprocessing, the first and second fluctuation removal processes areperformed in advance on the dynamic body region and the non-dynamic bodyregion detected by the dynamic body detecting unit 104, and the firstand second processed images obtained by such processes are reconfiguredin the dynamic body region setting unit 106. The similarity lies in thatthe image region set as the dynamic body region in the dynamic bodydetecting unit 104 is assumed as the dynamic body region for apredetermined time, and the first fluctuation removal process is notperformed on the subject set in the non-dynamic body region in thedynamic body detecting unit 104 although it is actually a dynamic body.Therefore, the image processing method shown in FIGS. 4 and 5 and theimage processing method described in the description of FIG. 2 areembodiments that can be realized based on the same technical idea.

First, FIG. 4 will be referenced. As shown in FIG. 4, the image inputunit 102 acquires the time-series images (S102). The dynamic bodydetecting unit 104 executes the dynamic body detecting process (S104).In this case, the difference in pixel value is calculated among aplurality of photographed images in the time-series images, and thedynamic body region and the non-dynamic body region are detected basedon the calculation result in the dynamic body detecting unit 104. Thefirst fluctuation removing unit 112 performs the first fluctuationremoval process on the non-dynamic body region detected by the dynamicbody detecting unit 104 (S106). In this case, the original image of thetime-series images acquired by the image input unit 102 and the firstprocessed image of after the first fluctuation removal process isperformed are held in the image processing apparatus 100. The dynamicbody region setting unit 106 resets the dynamic body region and thenon-dynamic body region (S108). The process executed in step S108 willbe hereinafter described with reference to FIG. 5.

The second fluctuation removing unit 114 performs the second fluctuationremoval process on the dynamic body region reset in step S108 (S110).Step S110 may be such that the second fluctuation removal process isperformed on all dynamic body regions, or may be such that the secondfluctuation removal process is performed only on the specified regionspecified through the operation input unit 110. The process such asnon-rigid body registration is a process of relatively large calculationamount. Thus, the calculation load is reduced by limiting the processingtarget to the specified region. If the dynamic body region not performedwith the second fluctuation process exists, the high image qualityprocessing unit 116 performs the high image quality processing on suchdynamic body region (S112). Regarding the dynamic body region performedwith the high image quality processing as well, the processing target ispreferably limited to a specified region from the standpoint of reducingthe calculation load.

The resetting method of the dynamic body region and the non-dynamic bodyregion corresponding to step S108 will now be described with referenceto FIG. 5. The process of step S108 is mainly realized using the dynamicbody region setting unit 106. The flow of processes shown in FIG. 5 isexecuted on all the pixels contained in the photographed image.

As shown in FIG. 5, whether or not the pixel PX(x,y) is of the dynamicbody is first determined (S132). The pixel PX(x,y) indicates anarbitrary pixel contained in the photographed image. Furthermore, (x,y)represents a coordinate position where the pixel PX(x,y) is arranged.The determination process in step S132 is executed based on theinformation of the dynamic body region and the non-dynamic body regiondetermined by the dynamic body detecting unit 104. If the dynamic bodyregion includes the pixel PX(x,y), the dynamic body region setting unit106 proceeds to the process of step S146. If the non-dynamic body regionincludes the pixel PX(x,y), the dynamic body region setting unit 106proceeds to the process of step S134.

In step S134, a time T of the timer 108 is decremented (S132). If time Tof the timer 108 is T=0, the time T=0 is maintained. The dynamic bodyregion setting unit 106 then determines whether or not the time T of thetimer 108 is zero (S136). If the time T of the timer 108 is T≠0, thedynamic body region setting unit 106 proceeds to the process of stepS148. If the time of the timer 108 is T=0, the dynamic body regionsetting unit 106 proceeds to the process of step S138. In step S138, thedynamic body region setting unit 106 outputs a pixel PX2(x,y) of thefirst processed image as the pixel PX(x,y) (S138).

The dynamic body region setting unit 106 then determines whether or nota next frame (photographed image) exists in the time-series images(S142). If the next frame exists, the dynamic body region setting unit106 proceeds to the process of step S144. If the next frame does notexist, the dynamic body region setting unit 106 terminates the series ofprocesses. In step S144, the dynamic body region setting unit 106references a photographed image corresponding to the next frame (S144).The dynamic body region setting unit 106 again executes the series ofprocesses starting from step S132 on the photographed image of the frameto reference.

If the process proceeds to the process of step S146 based on thedetermination process of step S132, the dynamic body region setting unit106 sets the time T of the timer 108 to a predetermined time T0 in stepS146 (S146). In step S148, the dynamic body region setting unit 106outputs the pixel PX1(x,y) of the original image not performed with thefirst fluctuation removal process as the pixel PX(x,y) (S148). Afterstep S148, the dynamic body region setting unit 106 proceeds to theprocess of step S142.

The above flow can be summarized as below. First, whether each pixel PXis of the dynamic body or of the non-dynamic body is determined in stepS132 based on the information of the dynamic body region and thenon-dynamic body region detected by the dynamic body detecting unit 104.The timer T of the timer 108 is set to a predetermined time T0 if thepixel PX is of the dynamic body. If the pixel PX is not of the dynamicbody, the time T of the timer 108 is decremented. That is, the dynamicbody region setting unit 106 counts the predetermined time T0 bydecrementing the time T of the timer 108 with the time point the pixelPX is determined as being of the dynamic body as a reference.

As described above, the dynamic body region setting unit 106 assumes theimage region as the dynamic body region for the predetermined time T0from when the dynamic body is detected. That is, the pixel PX determinedas being of the dynamic body in a certain frame is assumed as the pixelPX of the dynamic body in all the frames from the relevant frame to theframe of after the predetermined time T0. In order to realize suchassumption process, the pixel PX1 of the original image is output as thepixel PX until time T=0 after the time T of the timer 108 is set in theimage processing method.

In other words, whether or not time T=0 is determined in step S136, andthe pixel PX1 is output in the process of step S148 if not time T=0. Ifthe time T of the timer 108 becomes zero, the pixel PX2 of the firstprocessed image is output in step S138. In the image processing methodshown in FIG. 5, the resetting of the dynamic body region and thenon-dynamic body region is carried out by switching the image to outputfor every pixel using the first processed image obtained by performingthe first fluctuation removal process on the non-dynamic body region inadvance and the original image.

The flow of image processing according to the present embodiment hasbeen described above. In the present embodiment, the non-dynamic bodyregion is reset as the dynamic body region for a predetermined timeafter the dynamic body is detected, and adjustment is made such that thefirst fluctuation removal process is not performed on the reset dynamicbody region. As a result of such adjustment process, the firstfluctuation removal process can be avoided from being performed on theimage region that is actually a dynamic body but tends to be detected asthe non-dynamic body region, whereby the disappearance of the dynamicbody and the degradation of the image caused by the fluctuation removalprocess can be prevented. The image processing method described in thedescription of FIG. 2 includes resetting the dynamic body region and thenon-dynamic body region, and performing the first fluctuation removalprocess based on the reset information. Both methods include a technicalfeature common to the present embodiment of resetting the non-dynamicbody region as the dynamic body region for a predetermined time afterthe dynamic body is detected, whereby a significant effect in that thedisappearance of the dynamic body and the degradation of the imagecaused by the fluctuation removal process are prevented can be obtained.

[2-5: Regarding Method of Reducing Calculation Load by Block Division]

The method of reducing the calculation load by block dividing thephotographed image (hereinafter referred to as block division method)will now be described with reference to FIG. 6. FIG. 6 is an explanatoryview showing the method of reducing the calculation load by blockdividing the photographed image. As described earlier, the secondfluctuation removal process and the high image quality process areprocesses having high calculation load compared to the first fluctuationremoval process. Thus, in the present embodiment, such processes areallowed to be modified so as to be performed limited to the specifiedregion. The block division method shown in FIG. 6 divides thephotographed image to blocks of a predetermined size, and enables theblock to be specified as a specified region.

For instance, as shown in FIG. 6, the photographed image is divided into25 blocks. The user is presented with the photographed image, the firstprocessed image, or the second processed image (hereinafter referred toas display image) through the display screen. The boundary of the blocksmay be clearly shown or may not be clearly shown in the display image. Adynamic body block B1 including the dynamic body region and anon-dynamic body block B2 not including the dynamic body region may beclearly shown in the display image with color, pattern, or the like. Theuser specifies the block to perform the second fluctuation removalprocess or the high quality process through the operation input unit 110while referencing the display image.

When the block is specified, the second fluctuation removal process orthe high image quality process is performed on the dynamic body regionin the relevant block by the second fluctuation removing unit 114 or thehigh image quality processing unit 116. The dynamic body block B1performed with the second fluctuation removal process or the high imagequality process may be displayed in a different window. The calculationload of the image processing apparatus 100 thus can be reduced by beingconfigured such that the second fluctuation removal process or the highimage quality process is performed on the specified block. As a result,the function of the image processing apparatus 100 can be realizedwithout using an information processing apparatus mounted with anexpensive and high performance calculation processing apparatus.

[2-6: Regarding Example of Hardware Configuration of Image ProcessingApparatus 100]

The function of each component of the image processing apparatus 100 canbe realized using the hardware configuration of the informationprocessing apparatus shown in FIG. 7. For instance, the function of eachcomponent is realized by controlling the information processingapparatus shown in FIG. 7 using a computer program. The mode of theinformation processing apparatus shown here is arbitrary, and may be apersonal computer, portable telephone, portable information terminalsuch as PHS, PDA, video games, imaging device, or various informationhome electronics. PHS is an abbreviation for Personal Hand-phone System.PDA is an abbreviation for Personal Digital Assistant.

As shown in FIG. 7, the information processing apparatus mainly includesa CPU (Central Processing Unit) 902, a ROM (Read Only Memory) 904, a RAM(Random Access Memory) 906, a Host bus 908, a bridge 910, an externalbus 912, an interface 914, an input unit 916, an output unit 918, astorage unit 920, a drive 922, a connection port 924, and acommunication unit 926.

The CPU 902 functions as an arithmetic processing unit or a control unitand controls an entire operation of the constituent elements or some ofthe constituent elements based on various programs recorded on the ROM904, the RAM 906, the storage unit 920, or the removal recording medium928. The ROM 904 stores, for example, a program loaded on the CPU 902 ordata or the like used in an arithmetic operation. The RAM 906temporarily or perpetually stores, for example, a program loaded on theCPU 902 or various parameters or the like arbitrarily changed inexecution of the program. These constituent elements are connected toeach other by, for example, the host bus 908 which can performhigh-speed data transmission. The host bus 908, for example, isconnected to the external bus 912 in which a data transmission speed isrelatively low through the bridge 910.

The input unit 916 is, for example, an operation unit such as a mouse, akeyboard, a touch panel, button, a switch, or a lever. The input unit916 may be a remote control unit (so-called remote) that can transmit acontrol signal by using an infrared ray or other radio waves. The inputunit 916 includes an input control circuit or the like to transmitinformation input by using the operation unit to the CPU 902 through aninput signal.

The output unit 918 is, for example, a display device such as a CRT(Cathode Ray Tube), an LCD (Liquid Crystal Display), a PDP (PlasmaDisplay Panel), or an ELD (Electro-Luminescence Display), an audiooutput device such as a loudspeaker or headphones, a printer, a mobilephone, or a facsimile that can visually or auditorily notify a user ofacquired information.

The storage unit 920 is a device to store various data, and includes,for example, a magnetic storage device such as a hard disk drive (HDD;Hard Disk Drive), a semiconductor storage device, an optical storagedevice, or a magnetooptical storage device, or the like.

The drive 922 is a device that reads information recorded on the removalrecording medium 928 such as a magnetic disk, an optical disk, amagnetooptical disk, or a semiconductor memory or writes information inthe removal recording medium 928. The removal recording medium 928 is,for example, a DVD medium, a Blue-ray medium, an HD-DVD medium, acompact flash (CF; compactFlash) (registered trademark), a memorystick,or an SD memory card (Secure Digital memory card), or the like. As amatter of course, the removal recording medium 928 may be, for example,an IC card (Integrated Circuit Card) on which a non-contact IC chip ismounted, an electronic device, or the like.

The connection port 924 is a port such as an USB (Universal Serial Bus)port, an IEEE1394 port, an SCSI (Small Computer System Interface), anRS-232C port, or an optical audio terminal to which the externalconnection device 930 is connected. The external connection device 930is, for example, a printer, a mobile music player, a digital camera, adigital video camera, an IC recorder, or the like.

The communication unit 926 is a communication device to be connected toa network 932. For example, a communication card for a wired or wirelessLAN (Local Area Network), Bluetooth (registered trademark), or WUSB(Wireless USB), an optical communication router, an ADSL (AsymmetricDigital Subscriber Line) router, various communication modems, or thelike is used. The network 932 connected to the communication unit 926includes a wiredly or wirelessly connected network. For example, theInternet, a home-use LAN, infrared communication, broadcasting,satellite communication, or the like is used.

[2-7: Summary]

Lastly, the function configuration of the image processing apparatus ofthe present embodiment and the effects obtained from the functionconfiguration will be briefly summarized.

First, the function configuration of the image processing apparatusaccording to the present embodiment can be expressed in the followingmanner. The image processing apparatus includes a dynamic body detectingunit, a dynamic body region setting unit, and a fluctuation removalprocessing unit. The dynamic body detecting unit detects a dynamic bodycontained in the moving image. The dynamic body detecting unit comparesthe pixel values of a plurality of frames configuring the moving image,and when the amount of difference of the pixel value is greater than orequal to a predetermined amount, determines the image regioncorresponding to the relevant pixel value as the region of the dynamicbody. That is, the dynamic body region is detected by the dynamic bodydetecting process.

During a predetermined time from the time point the dynamic body isdetected by the dynamic body detecting unit, the dynamic body regionsetting unit sets the region containing the dynamic body at the relevantdetection time point as the dynamic body region. That is, with respectto the image region corresponding to the dynamic body region detected inthe dynamic body detecting process by the dynamic body detecting unit,the image region is reset as the dynamic body region even if such imageregion becomes the non-dynamic body region after the dynamic body hasmoved. As a result, even if the portion that is less likely to bedetected as the dynamic body through the dynamic body detecting processby the dynamic body detecting unit is contained in one part of thedynamic body, such one part of the dynamic body can be avoided frombeing set as the non-dynamic body region.

The fluctuation removal processing unit performs the fluctuation removalprocess on the region other than the dynamic body region set by thedynamic body region setting unit. The fluctuation removal processperformed by the fluctuation removal processing unit corresponds to thefirst fluctuation removal process. It can be recognized that thefluctuation removal process performed by the fluctuation removalprocessing unit may be the fluctuation removal process other than thefirst fluctuation removal process. If the fluctuation removal processperformed by the fluctuation removal processing unit is the firstfluctuation removal process, the dynamic body disappears or the imagedegrades if the relevant fluctuation removal process is performed on theimage region containing the dynamic body.

However, the fluctuation removal processing unit performs thefluctuation removal process on the image region other than the dynamicbody region set by the dynamic body region setting unit. Thus, one partof the dynamic body that is actually the dynamic body but is less likelyto be detected as the dynamic body in the process of the dynamic bodydetecting unit is masked so that the fluctuation removal process is notperformed thereon. As a result, disappearance of part of the dynamicbody or degradation of the image due to the fluctuation removal processis avoided, and a satisfactory processed image is obtained.

The fluctuation removal processing unit may be configured to performother fluctuation removal process, in which the video of the dynamicbody contained in the dynamic body region does not disappear, on part ofor the entire dynamic body region set by the dynamic body region settingunit. According to such configuration, (other) fluctuation removalprocess is performed on the image region that is actually a non-dynamicbody region but is assumed as the dynamic body region by the maskingprocess of the dynamic body region setting unit and thus not performedwith the fluctuation removal process. As a result, the image quality ofthe processed image can be more significantly enhanced.

The image processing apparatus may also include a super-resolutionprocessing unit for performing the super-resolution process on part ofor the entire dynamic body region not performed with the otherfluctuation removal process by the fluctuation removal processing unitof the dynamic body region set by the dynamic body region setting unit.As described above, the other fluctuation removal process may not beexecuted on all the dynamic body regions. For instance, the otherfluctuation removal process may be performed only on the image regionthat is set as the dynamic body region by the dynamic body regionsetting unit but actually does not include the dynamic body, and thesuper-resolution process may be performed on the dynamic body regionthat actually contains the dynamic body.

As described above, the dynamic body region setting unit sets the imageregion corresponding to the dynamic body region detected in the firstframe as the dynamic body region regardless of the presence of thedynamic body with respect to all the frames from the frame at the timepoint the dynamic body is detected to the frame of after elapse of apredetermined time. The optimum process is sometimes preferably dividedfor the dynamic body region containing the dynamic body and the dynamicbody region not containing the dynamic body in the dynamic body regions.In particular, a processed image of a more natural image quality isobtained by enabling the user to specify the dynamic body region to beperformed with the other fluctuation removal process and the dynamicbody region to be performed with the super-resolution process. From thestandpoint of calculation load, the calculation load can be effectivelydispersed by appropriately changing the processing content instead ofperforming the same process on all the dynamic body regions.

The fluctuation removal process performed on all regions of the movingimage excluding the dynamic body region by the fluctuation removalprocessing unit is the averaging process or the median process of thepixel value contained in each frame of the moving image. The fluctuationremoval process performed on the dynamic body region is a fluctuationremoval process based on the non-rigid body registration, and the like.

As described above, the dynamic body disappears or the image qualitydegrades if the averaging process or the median process is performed onthe dynamic body region. According to the fluctuation removal processbased on the non-rigid body registration, the dynamic body does notdisappear and the image quality does not degrade. However, thecalculation amount of the non-rigid body registration process is verylarge compared to the averaging process or the median process. Thus, theprocessing target of the filtering process such as the averaging processor the median process, and the process such as the non-rigid bodyregistration is to be appropriately divided.

In the image processing apparatus described above, the dynamic bodyregion and the non-dynamic body region are set by the dynamic bodydetecting unit and the dynamic body region setting unit, and thefluctuation removal process is divided for each region. In such case,the averaging process or the median process, and the non-rigid bodyregistration process can be assigned in the above manner, so that theimage quality of the moving image can be effectively enhanced whilesuppressing the increase of the calculation load.

The image processing method by the image processing apparatus includes adynamic body detecting step, a dynamic body region setting step, and afluctuation removal processing step. In the dynamic body detecting step,the dynamic body contained in the moving image is detected. In thedynamic body region setting step, during a predetermined time from thetime point when the dynamic body is detected in the dynamic bodydetecting step, the region containing the dynamic body at the relevantdetection time point is set as the dynamic body region. In thefluctuation removal processing step, the fluctuation removal process isperformed on the entire region of the moving image excluding the dynamicbody region set in the dynamic body region setting step. When each framecontained in the moving image is processed in the processing steps insuch manner, the disappearance of the dynamic body or the degradation ofthe image quality of the dynamic body portion due to the fluctuationremoval process can be avoided.

The image processing method is realized by using a program for causing acomputer to realize a dynamic body detecting function, a dynamic bodyregion setting function, and a fluctuation removal processing function.In the dynamic body detecting function, the dynamic body contained inthe moving image is detected. The dynamic body region setting functionsets, during a predetermined time from the time point when the dynamicbody is detected by the dynamic body detecting function, the region thatcontained the dynamic body at the relevant detection time point as thedynamic body region. The fluctuation removal processing function is afunction of performing the fluctuation removal process on the entireregion of the moving image excluding the dynamic body region set by thedynamic body region setting function. When each frame contained in themoving image is processed using the program having such functions, thedisappearance of the dynamic body or the degradation of the imagequality of the dynamic body portion due to the fluctuation removalprocess can be avoided.

(Remark)

The first fluctuation removing unit 112 and the second fluctuationremoving unit 114 serve as the fluctuation removal processing unit. Thehigh image quality processing unit 116 serves as the super-resolutionprocessing unit. The averaging filtering process and the time medianfiltering process serve as the fluctuation removal process. Thenon-rigid body registration process serves as the other fluctuationremoval process.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For instance, in the above-described embodiment, description has beenmade as if the mode of the image processing apparatus 100 is theinformation processing apparatus. However, the image processingapparatus 100 according to the embodiment may be configured as part ofan imaging device. In this case, the image processing apparatus 100includes an optical system to which the light reflected by the subjectenters, and a photoelectric conversion element for photoelectricconverting the light entered through the optical system. The imaged dataconverted to an electric signal by the photoelectric conversion elementis input to the image input unit 102 as a photographed image. An A/Dconverter may be arranged in the pre-stage of the image input unit 102,so that the imaged data input as an analog signal is digitalized, andthen input to the image input unit 102. According to such configuration,the image processing apparatus 100 can function as the imaging device.

What is claimed is:
 1. An image processing apparatus comprising: adynamic body detecting unit for detecting a dynamic body contained in amoving image; a dynamic body region setting unit for, during apredetermined time from a time point the dynamic body is detected by thedynamic body detecting unit, setting a region containing the dynamicbody at the detection time point as a dynamic body region; and afluctuation removable processing unit for performing a fluctuationremoval process on a region other than the dynamic body region set bythe dynamic body region setting unit.